Content server media stream management

ABSTRACT

Mechanisms are provided to manage media stream transmissions at a content server. A content server detects that a user on a device such as a mobile device has stopped playing a live media stream. The content server maintains information associating the user with the media stream and time information. When the content server detects that a user wishes to resume playing a media stream, the user can continue viewing the media stream from where stoppage occurred. In many instances, the content server stores many hours of live media stream data and allows a user to select a particular starting point.

CROSS REFERENCE To RELATED PATENT APPLICATION

This application claims priority and is a continuation of U.S. patentapplication Ser. No.14/298,491, entitled “CONTENT SERVER MEDIA STREAMMANAGEMENT,” filed Jun. 6, 2014 by Kent Karlsson et al., which is acontinuation of U.S. patent application Ser. No. 12/345,534, entitled“CONTENT SERVER MEDIA STREAM MANAGEMENT,” filed Dec. 29, 2008 by KentKarlsson et al., now Issued Pat. No. 8,782,275 on Jul. 15, 2014, whichis a continuation of U.S. patent application Ser. No. 12/101,913,entitled “CONTENT SERVER MEDIA STREAM MANAGEMENT,” filed April 11, 2008by Kent Karlsson et al., now Issued U.S. Pat. No. 9,003,051 on Apr. 7,2015, all of which is incorporated in its entirety by this reference forall purposes.

TECHNICAL FIELD

The present disclosure relates to managing and manipulating mediastreams.

DESCRIPTION OF RELATED ART

Content servers provide media streams to individual devices usingprotocols such as the Real-Time Transport Protocol (RTP). A separatesession is used to carry each content stream such as a video or audiostream. RTP specifies a standard packet format that is used to carryvideo and audio data such as Moving Pictures Expert Group (MPEG) videodata including MPEG-2 and MPEG-4 video frames. In many instances,multiple frames are included in a single RTP packet. The MPEG framesthemselves may be reference frames or may be frames encoded relative toa reference frame.

Conventional techniques and mechanisms allow a content server totransmit a media stream to a client device. However, mechanisms formanaging and manipulating the media stream efficiently and effectivelyare limited. Consequently, it is desirable to provide improvedtechniques and mechanisms for transmitting media streams from contentservers to client devices.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure may best be understood by reference to the followingdescription taken in conjunction with the accompanying drawings, whichillustrate particular embodiments.

FIG. 1 illustrates an exemplary system for use with embodiments of thepresent invention.

FIG. 2 illustrates one example of a Real-Time Transport Protocol (RTP)packet.

FIG. 3 illustrates one example of an RTP stream.

FIG. 4 illustrates one example of content server having channel buffers.

FIG. 5 illustrates one example of a media stream management table.

FIG. 6 illustrates example of entry points in a media stream.

FIG. 7 illustrates one example of a technique for resuming mediatransmissions.

FIG. 8 illustrates one example of a system for processing media streams.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Reference will now be made in detail to some specific examples of theinvention including the best modes contemplated by the inventors forcarrying out the invention. Examples of these specific embodiments areillustrated in the accompanying drawings. While the invention isdescribed in conjunction with these specific embodiments, it will beunderstood that it is not intended to limit the invention to thedescribed embodiments. On the contrary, it is intended to coveralternatives, modifications, and equivalents as may be included withinthe spirit and scope of the invention as defined by the appended claims.

For example, the techniques of the present invention will be describedin the context of the Real-Time Transport Protocol (RTP) and theReal-Time Streaming Protocol (RTSP). However, it should be noted thatthe techniques of the present invention apply to a variations of RTP andRTSP. In the following description, numerous specific details are setforth in order to provide a thorough understanding of the presentinvention. Particular example embodiments of the present invention maybe implemented without some or all of these specific details. In otherinstances, well known process operations have not been described indetail in order not to unnecessarily obscure the present invention.

Various techniques and mechanisms of the present invention willsometimes be described in singular form for clarity. However, it shouldbe noted that some embodiments include multiple iterations of atechnique or multiple instantiations of a mechanism unless notedotherwise. For example, a system uses a processor in a variety ofcontexts. However, it will be appreciated that a system can use multipleprocessors while remaining within the scope of the present inventionunless otherwise noted. Furthermore, the techniques and mechanisms ofthe present invention will sometimes describe a connection between twoentities. It should be noted that a connection between two entities doesnot necessarily mean a direct, unimpeded connection, as a variety ofother entities may reside between the two entities. For example, aprocessor may be connected to memory, but it will be appreciated that avariety of bridges and controllers may reside between the processor andmemory. Consequently, a connection does not necessarily mean a direct,unimpeded connection unless otherwise noted.

Overview

Mechanisms are provided to manage media stream transmissions at acontent server. A content server detects that a user on a device such asa mobile device has stopped playing a live media stream. The contentserver maintains information associating the user with the media streamand time information. When the content server detects that a user wishesto resume playing a media stream, the user can continue viewing themedia stream from where stoppage occurred. In many instances, thecontent server stores many hours of live media stream data and allows auser to select a particular starting point.

Example Embodiments

A variety of mechanisms are used to deliver media streams to devices. Inparticular examples, a client establishes a session such as a Real-TimeStreaming Protocol (RTSP) session. A server computer receives aconnection for a media stream, establishes a session, and provides amedia stream to a client device. The media stream includes packetsencapsulating frames such as Moving Pictures Expert Group (MPEG) frames.The MPEG frames themselves may be key frames or differential frames. Thespecific encapsulation methodology used by the server depends on thetype of content, the format of that content, the format of the payload,the application and transmission protocols being used to send the data.After the client device receives the media stream, the client devicedecapsulates the packets to obtain the MPEG frames and decodes the MPEGframes to obtain the actual media data.

In many instances, a server computer obtains media data from a varietyof sources, such as media libraries, cable providers, satelliteproviders, and processes the media data into MPEG frames such as MPEG-2or MPEG-4 frames. In particular examples, a server computer may encodesix media streams of varying bit rates for a particular channel fordistribution to a variety of disparate devices.

A user on a device such as a mobile device obtains a media stream byestablishing an RTSP session. According to various embodiments, a usermay stop viewing the media stream at some point. In some examples, auser may manually stop viewing a media stream or may want to switch fromviewing on a mobile phone to viewing the media stream on a computer ortelevision. In typical implementations, stopping a media stream closesan RTSP session. When a user resumes viewing the media stream, the usermay have missed a large portion of a program due to time lapse. A usermay have a digital video recorder or a personal video recorder, but therecorder may require configuration. Devices such as mobile devices alsodo not typically have recording capabilities.

Consequently, the techniques and mechanisms of the present inventionprovide the user with the ability to resume a media stream at some latertime. For example, a user stops viewing a video stream at a particulartime. The content server continues to buffer the live media stream andmaintains what the user was viewing and when viewing stopped. When auser attempts to reconnect, either on the mobile device or on some otherdevice, the content server can continue sending the media stream wherethe user left off. In some instances, the content server gives the userthe option to play the media stream as a live stream or to play themedia stream where the user stopped viewing.

According to various embodiments, the content server buffers thousandsof media streams and can allow a user to view media stream content fromany starting point still within the content server buffers. The user maybe able to request not only a particular channel, but a particular time.A user can watch recorded content without having to configure arecording in advance on all devices including devices that do not haverecording capabilities. In particular embodiments, the user can not skipover commercials as the content server manages and stores the mediastreams.

FIG. 1 is a diagrammatic representation illustrating one example of asystem that can use the techniques and mechanisms of the presentinvention. According to various embodiments, content servers 119, 121,123, and 125 are configured to provide media content to a mobile device101 using protocols such as RTP and RTCP. Although a mobile device 101is shown, it should be recognized that other devices such as set topboxes and computer systems can also be used. In particular examples, thecontent servers 119, 121, 123, and 125 can themselves establish sessionswith mobile devices and stream video and audio content to mobiledevices. However, it is recognized that in many instances, a separatecontroller such as controller 105 or controller 107 can be used toperform session management using a protocol such as RTSP. It isrecognized that content servers require the bulk of the processing powerand resources used to provide media content mobile devices. Sessionmanagement itself may include far fewer transactions. Consequently, acontroller can handle a far larger number of mobile devices than acontent server can. In some examples, a content server can operatesimultaneously with thousands of mobile devices, while a controllerperforming session management can manage millions of mobile devicessimultaneously.

By separating out content streaming and session management functions, acontroller can select a content server geographically close to a mobiledevice 101. It is also easier to scale, as content servers andcontrollers can simply be added as needed without disrupting systemoperation. A load balancer 103 can provide further efficiency duringsession management using RTSP 133 by selecting a controller with lowlatency and high throughput.

According to various embodiments, the content servers 119, 121, 123, and125 have access to a campaign server 143. The campaign server 143provides profile information for various mobile devices 101. In someexamples, the campaign server 143 is itself a content server or acontroller. The campaign server 143 can receive information fromexternal sources about devices such as mobile device 101. Theinformation can be profile information associated with various users ofthe mobile device including interests and background. The campaignserver 143 can also monitor the activity of various devices to gatherinformation about the devices. The content servers 119, 121, 123, and125 can obtain information about the various devices from the campaignserver 143. In particular examples, a content server 125 uses thecampaign server 143 to determine what type of media clips a user on amobile device 101 would be interested in viewing.

According to various embodiments, the content servers 119, 121, 123, and125 are also receiving media streams from content providers such assatellite providers or cable providers and sending the streams todevices using RTP 131. In particular examples, content servers 119, 121,123, and 125 access database 141 to obtain desired content that can beused to supplement streams from satellite and cable providers. In oneexample, a mobile device 101 requests a particular stream. A controller107 establishes a session with the mobile device 101 and the contentserver 125 begins streaming the content to the mobile device 101 usingRTP 131. In particular examples, the content server 125 obtains profileinformation from campaign server 143.

In some examples, the content server 125 can also obtain profileinformation from other sources, such as from the mobile device 101itself. Using the profile information, the content server can determinewhether a client device would support a burst of data. For example, thecontent server could determine that the client device has a particularbuffer size and reports when the buffer is low or empty. When the clientdevice supports buffer bursts, the content server can transmit availabledata at a higher bit rate to the client device when the client buffer islow or empty. In some instances, a content server buffer for aparticular channel will be empty and nothing can be transmitted to theclient. However, if the content server buffer for the particular channelhas data available, a burst of data can be transmitted to replenish theclient buffer. In a particular example, 8 seconds of video data istransmitted in a short amount of time. Extra packets can simply betransmitted. However, a lower quality stream selected from the samechannel or from a database can also be transmitted to replenish theclient buffer.

FIG. 2 illustrates one example of an RTP packet. An RTP packet 201includes a header 211. According to various embodiments, the header 211includes information such as the version number, amount of padding,protocol extensions, application level, payload format, etc. The RTPpacket 201 also includes a sequence number 213. Client applicationsreceiving RTP packets expect that the sequence numbers for receivedpackets be unique. If different packets have the same sequence number,erroneous operation can occur. RTP packets also have a timestamp 215that allows jitter and synchronization calculations. Fields 217 and 219identify the synchronization source and the contributing source.Extensions are provided in field 221.

According to various embodiments, data 231 holds actual media data suchas MPEG frames. In some examples, a single RTP packet 201 holds a singleMPEG frame. In many instances, many RTP packets are required to hold asingle MPEG frame. In instances where multiple RTP packets are requiredfor a single MPEG frame, the sequence numbers change across RTP packetswhile the timestamp 215 remains the same across the different RTPpackets. Different MPEG frames include I-frames, P-frames, and B-frames.I-frames are intraframes coded completely by itself. P-frames arepredicted frames which require information from a previous I-frame orP-frame. B-frames are bi-directionally predicted frames that requireinformation from surrounding I-frames and P-frames.

Because different MPEG frames require different numbers of RTP packetsfor transmission, two different streams of the same time duration mayrequire different numbers of RTP packets for transmission. Simplyreplacing a clip with another clip would not work, as the clips may havedifferent numbers of RTP packets and having different impacts on thesequence numbers of subsequent packets.

FIG. 3 illustrates one example of an RTP packet stream. An RTP packetstream 301 includes individual packets having a variety of fields andpayload data. According to various embodiments, the fields include atimestamp 303, sequence 505, marker 307, etc. The packets also includepayload data 309 holding MPEG frames such as I, P, and B-frames.Timestamps for different packets may be the same. In particularexamples, several packets carrying portions of the same I-frame have thesame time stamp. However, sequence numbers are different for eachpacket. Marker bits 307 can be used for different purposes, such assignaling the starting point of an advertisement.

According to various embodiments, packets with sequence numbers 4303,4304, and 4305 carrying potions of the same I-frame and have the sametimestamp of 6. Packets with sequence numbers 4306, 4307, 4308, and 4309carry P, B, P, and P-frames and have timestamps of 7, 8, 9, and 10respectively. Packets with sequence numbers 4310 and 4311 carrydifferent portions of the same I-frame and both have the same timestampof 11. Packets with sequence numbers 4312, 4313, 4314, 4315, and 4316carry P, P, B, P, and B-frames respectively and have timestamps 12, 13,14, 15, and 16. It should be noted that the timestamps shown in FIG. 3are merely representational. Actual timestamps can be computed using avariety of mechanisms.

For many audio encodings, the timestamp is incremented by thepacketization interval multiplied by the sampling rate. For example, foraudio packets having 20 ms of audio sampled at 8,000 Hz, the timestampfor each block of audio increases by 160. The actual sampling rate mayalso differ slightly from this nominal rate. For many video encodings,the timestamps generated depend on whether the application can determinethe frame number. If the application can determine the frame number, thetimestamp is governed by the nominal frame rate. Thus, for a 30 f/svideo, timestamps would increase by 3,000 for each frame. If a frame istransmitted as several RTP packets, these packets would all bear thesame timestamp. If the frame number cannot be determined or if framesare sampled a periodically, as is typically the case for softwarecodecs, the timestamp may be computed from the system clock.

While the timestamp is used by a receiver to place the incoming mediadata in the correct timing order and provide playout delay compensation,the sequence numbers are used to detect loss. Sequence numbers increaseby one for each RTP packet transmitted, timestamps increase by the time“covered” by a packet. For video formats where a video frame is splitacross several RTP packets, several packets may have the same timestamp.For example, packets with sequence numbers 4317 and 4318 have the sametimestamp 17 and carry portions of the same I-frame.

FIG. 4 illustrates one example of content server buffers. According tovarious embodiments, devices 401, 411, 421, and 431 have individualbuffers 403, 413, 423, and 433. Content server 451 includes channelbuffers 453, 455, 457, 459, and 461. In particular embodiments, acontent server 451 detects that a device has little or no data remainingin a buffer. A device such as a mobile device may have little or no datain a buffer when network conditions cause transmission delays and droppackets or when a device initially requests a media stream. To improveuser experience, a content server 451 bursts available data for arequested stream to a device 411 having an low or empty buffer. In someexamples, the content server 451 transmits data from channel buffer 455to device 411 at double the usual transmission bit rate for a fixednumber of seconds.

In other examples, the content server 451 transmits data from a lowquality stream in channel buffer 453 to device 411. Transmitting a lowerquality stream allows a buffer to be filled while maintaining the sametransmission bit rate. For example, a stream in channel buffer 453 maybe a 50 mbps stream while a stream in channel buffer 455 may be a 100mbps stream. More frames from the lower quality stream can betransmitted to allow the device 411 to resume playback with decreaseddelay.

According to various embodiments, content server buffers may or may notbe prefilled. In some examples, once a media stream has been requested,the corresponding channel buffer is filled at the content server.However, channel buffers corresponding to media streams not yetrequested are typically not prefilled or prewarmed. Playback can bedelayed while the content server channel buffers are filled.Consequently, the techniques and mechanisms of the present inventioncontemplate prefilling channel buffers. According to variousembodiments, the content server channel buffers are prefilled using livestreams from cable and satellite providers and continually refreshedwith the most recent streaming data. In some instances, all channelbuffers are prefilled. In other instances, selected channel buffers areprefilled and refreshed using satellite and cable media streams.

FIG. 5 illustrates one example of a mechanism for maintaining timinginformation. According to various embodiments, a content servermaintains information about the program a user last watched and when theuser stopped watching the program. In particular embodiments, a mediastream management database 501 holds user identifiers 511, 521, 531,541, 551, and 561. The user identifiers may correspond to specificsubscribers, individuals, or devices. In particular embodiments, themedia stream management database 501 also holds information about themedia stream being viewed by a particular user. The media stream fields513, 523, 533, 543, 553, and 563 identify particular channels orparticular media streams that a user stopped viewing. Timing markers515, 525, 535, 545, 555, and 565 identify points in time where viewingstopped. In some instances, the timing markers correspond to timeinformation included in RTP packets.

According to various embodiments, a content server detects a stoppageevent, such as a session close message, a session break message, orother disruption in the transmission of the stream to a device, andmarks the time when the viewing stopped. In some examples, the stoppagepoint is recorded as being a few seconds before an actual stoppage eventis detected, to allow some possible overlap when a user resumes viewingor listening to a stream.

FIG. 6 illustrates various starting points at which a media stream canresume in an RTP packet stream. An RTP packet stream 601 includesindividual packets having a variety of fields and payload data.According to various embodiments, the fields include a timestamp 603,sequence 605, marker 607, etc. The packets also include payload data 609holding MPEG frames such as I, P, and B-frames. Timestamps for differentpackets may be the same. In particular examples, several packetscarrying portions of the same I-frame have the same time stamp. However,sequence numbers are different for each packet. Marker bits 607 can beused for different purposes, such as signaling the starting point of anadvertisement.

According to various embodiments, packets with sequence numbers 4303,4304, and 4305 carrying potions of the same I-frame and have the sametimestamp of 6. Packets with sequence numbers 4306, 4307, 4308, and 4309carry P, B, P, and P-frames and have timestamps of 7, 8, 9, and 10respectively. Packets with sequence numbers 4310 and 4311 carrydifferent portions of the same I-frame and both have the same timestampof 11. Packets with sequence numbers 4312, 4313, 4314, 4315, and 4316carry P, P, B, P, and B-frames respectively and have timestamps 12, 13,14, 15, and 16. It should be noted that the timestamps shown in FIG. 6are merely representational. Actual timestamps can be computed using avariety of mechanisms.

According to various embodiments, positions 611, 613, and 615 indicatepoints at which a media stream may be resumed. In particularembodiments, a media stream resumes at an I-frame to provide a devicewith enough information to generate a complete picture immediately,instead of providing a P-frame or a B-frame with only differentialinformation insufficient to generate a complete picture. Positions 611,613, and 615 or corresponding timestamps 6, 11, and 17 may be associatedwith a user device and maintained at a content server. According tovarious embodiments, possible starting positions may include times manyhours or many days in the past, depending on the amount of buffer spaceat a content server.

FIG. 7 illustrates one technique for performing for resuming mediastream transmissions. In many conventional implementations, when a mediastream session is broken for any reason, the user must reestablish thesession and the media stream transmissions resume at a current point intime. In some examples, a user may miss important content simply byswitching devices or losing wireless reception. Consequently, thetechniques of the present invention allow a user to resume a mediastream where the stream was stopped. In some examples, a user can electwhether to play a stream as a live stream or play a stream where thestream was stopped. In still other examples, a user can elect a specifictime to view a stream.

According to various embodiments, an indication that a media stream isno longer being played is received at 701. In particular embodiments,the indication is a session break message. In other embodiments, theindication a specialized command or some other termination of service.At 703, an entry is maintained associating the user with the mediastream being viewed and a time indicator. According to variousembodiments, a device identifier is maintained instead of a useridentifier. Channels and quality levels may also be optionallymaintained. At 707, an indication is received that a user is resumingplayback. At 711, the media stream identifier and time indicatorinformation for the user is obtained. At 713, the media streamappropriate for the user and device is located at the particular timeindicator. According to various embodiments, a high bandwidth mediastream is obtained if the user is viewing the media stream using largescreen device connected to a high speed network. In particularembodiments, a low bandwidth media stream is obtained if the user isviewing the media stream using a small screen device connected to abandwidth limited network. At 715, the media stream is transmitted tothe user.

FIG. 8 illustrates one example of a content server. According toparticular embodiments, a system 800 suitable for implementingparticular embodiments of the present invention includes a processor801, a memory 803, an interface 811, and a bus 815 (e.g., a PCI bus orother interconnection fabric) and operates as a streaming server. Whenacting under the control of appropriate software or firmware, theprocessor 801 is responsible for modifying and transmitting live mediadata to a client. Various specially configured devices can also be usedin place of a processor 801 or in addition to processor 801. Theinterface 811 is typically configured to end and receive data packets ordata segments over a network.

Particular examples of interfaces supports include Ethernet interfaces,frame relay interfaces, cable interfaces, DSL interfaces, token ringinterfaces, and the like. In addition, various very high-speedinterfaces may be provided such as fast Ethernet interfaces, GigabitEthernet interfaces, ATM interfaces, HSSI interfaces, POS interfaces,FDDI interfaces and the like. Generally, these interfaces may includeports appropriate for communication with the appropriate media. In somecases, they may also include an independent processor and, in someinstances, volatile RAM. The independent processors may control suchcommunications intensive tasks as packet switching, media control andmanagement.

According to various embodiments, the system 800 is a content serverthat also includes a transceiver, streaming buffers, and a program guidedatabase. The content server may also be associated with subscriptionmanagement, logging and report generation, and monitoring capabilities.In particular embodiments, functionality for allowing operation withmobile devices such as cellular phones operating in a particularcellular network and providing subscription management. According tovarious embodiments, an authentication module verifies the identity ofdevices including mobile devices. A logging and report generation moduletracks mobile device requests and associated responses. A monitor systemallows an administrator to view usage patterns and system availability.According to various embodiments, the content server 891 handlesrequests and responses for media content related transactions while aseparate streaming server provides the actual media streams.

Although a particular content server 891 is described, it should berecognized that a variety of alternative configurations are possible.For example, some modules such as a report and logging module 853 and amonitor 851 may not be needed on every server. Alternatively, themodules may be implemented on another device connected to the server. Inanother example, the server 891 may not include an interface to anabstract buy engine and may in fact include the abstract buy engineitself. A variety of configurations are possible.

In the foregoing specification, the invention has been described withreference to specific embodiments. However, one of ordinary skill in theart appreciates that various modifications and changes can be madewithout departing from the scope of the invention as set forth in theclaims below. Accordingly, the specification and figures are to beregarded in an illustrative rather than a restrictive sense, and allsuch modifications are intended to be included within the scope ofinvention.

1. A method comprising: maintaining a plurality of channel buffers at acontent server, the content server operable to provide a plurality ofmedia streams to a plurality of devices, wherein the plurality ofchannel buffers are prefilled and refreshed with the plurality of mediastreams including first media content encoded at a plurality ofdifferent quality levels; receiving a first request for first mediacontent from a user, the first request identifying a first bandwidth anda first position in the first media content; selecting a first mediastream from the plurality of media streams by using first bandwidthcharacteristics; transmitting the first media stream to the userbeginning with packets having timestamps corresponding to the firstposition; receiving a second request for first media content from theuser, the second request identifying a second bandwidth and a secondposition in the first media content; selecting a second media streamfrom the plurality of media streams by using second bandwidthcharacteristics; transmitting the second media stream to the userbeginning with packets having timestamps corresponding to the secondposition, wherein the user makes the first request from a first deviceand the second request from a second device different from the firstdevice.